IN THE CLAIMS: 



Please amend Claims 1, 4, 20, 23, and 39 as follows. 

1 . (Currently Amended) A simulator apparatus with which an operator plays a 
simulation in mixed reality space including a virtual space and real space in which a real 
object(s) is placed, said simulator comprising: 

a viewpoint detection unit adapted to detect the position/orientation of a viewpoint 
of the operator; 

an inputting unit adapted to input a real space image corresponding to the 
position/orientation of the viewpoint of the operator; 

a geometric information acquisition unit adapted to acquire geometric information 
of the real object(s) placed in the real space; 

a rule memory adapted to store rules for controlling the action patterns of virtual 

object(s); 

a status memorv adapted to store a simulation progress status: 
a computation unit adapted to determine the next action pattern of the virtual 
object(s) as if a virtual object had its own will by rcfeiring to said ml c mcmoiy based on a 
r e lation among the position/ori e ntation of the viewpoint of the ope r ator, position(s) of the virtual 
object(^s) and the g e ometric information of the real objcct(s) based on a command input by the 
operator, the simulation progress status, the rules and the geometric information of the real 
obiect(s) : 
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a virtual space generation unit adapted to generate a virtual space image on the 
basis of the a next position/orientation of the virtual object(s) after according to the determined 
action pattern and the position/orientation of the viewpoint of the operator; and 

a mixed reality image generation unit adapted to generate a mixed reality space 
image by superimposing or overlaying the virtual space image on the real space image^ 

wherein the next position/orientation of the virtual object(s) is determined based 
on a location relationship among the position/orientation of the viewpoint of the operator, 
position(s) of the virtual object(s) and the geometric information of the real object(s) , 

2. (Previously Presented) The apparatus according to claim 1, wherein said 
inputting unit captures real space images of said operator's view of the real space, 

and the apparatus further comprises a video see-through type display that the 
operator wears wherein the mixed reality images are displayed. 



3. (Previously Presented) The apparatus according to claim 1, further comprising 
an optical see-through type display that the operator wears wherein said virtual space image is 
displayed. 



4. (Currently Amended) The apparatus according to claim 1, further comprising: 
a status detector that detects a status of the operator, 

wherein said computation unit also determines a next action pattern of the virtual 
object in accordance with the status of the operator, i-ul c sto r ed in said rule mcmoiy and in 
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coiTCspondenc c with the position/orientation of th e real object aiid/or th e status of the operator, 
and com p utes a positioii/oriciitation of the virtual obj e ct after the det c miincd action . 



Claims 5-7 (Canceled) 



8. (Previously Presented) The apparatus according to claim 1, wherein the real 
object(s) include other operators who operate said simulator apparatus, and the other operators 
share a single mixed reality space with the operator. 



9. (Previously Presented) The apparatus according to claim 1, wherein the real 
object is an object which is fixed in position in the real space, and 

said geometric information acquisition unit comprises: 

a predetermined memory for pre-storing position information and shape 
information of the real object; and 

a reading unit that reads out the position information and shape information of the 
real object from said predetermined memory as needed. 



10. (Previously Presented) The apparatus according to claim 1, wherein the real 
object is an object which is movable but does not deform, and 

said geometric information acquisition unit comprises: 

a predetermined memory for pre-storing shape information of the real object; 
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a position/orientation sensor for detecting a position/orientation of the real object; 
and 

a setting unit that sets a region the real object is expected to occupy in the 
mixed reality space in accordance with the detected position/orientation of the real object. 

Claims 11-17 (Canceled) 

18. (Previously Presented) The apparatus according to claim 1, wherein said 
viewpoint detection unit detects a position/orientation of the head of the operator, and said 
apparatus further comprises: 

a detector that detects a position/orientation of a hand of the operator; and 
a recognition unit adapted to recognize a relative position of the hand of the 
operator with respect to the head as a command on the basis of an output from said detector. 

19. (Previously Presented) The apparatus according to claim 1, wherein said 
virtual space generation unit comprises: 

an alignment unit that aligns the position/orientation of the real object to the 
position/orientation of the virtual object after movement; 

a generation unit that generates an image of the virtual object after alignment in 
correspondence with an occlusion relationship; and 

a head-mounted display device. 
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20. (Currently Amended) An image processing method for a simulator apparatus 
with which an operator plays a game in a mixed reahty space in which a real object(s) is placed, 
comprising: 

a viewpoint detection step for detecting the position/orientation of a viewpoint of 

the operator; 

an inputting step for inputting a real space image corresponding to the 
position/orientation of the viewpoint of the operator; 

a geometric information acquisition step for acquiring geometric information of 
the real object(s) placed in the real space; 

a computation step for determining the next action pattern of virtual object(s) as if 
a virtual object had its own will by rcfcning to rul e s for controlling the action of th e virtual 
obj c ct(s)bas e d on a relation among the position/ori e ntation of the viewpoint of the operator, 
positiQn(s) of the virtual object(s) and th e geometric information of th e r e al object(s) based on a 
command input by the operator, a simulation progress status stored in a status memory, rules 
stored in a rule memory for controlling the action patterns of the virtual obiectfs) and the 
geometric information of the real obiect(s) : 

a virtual space generation step for generating a virtual space image on the basis of 
the a next position/orientation of the virtual object(s) after according to the determined action 
pattern and the position/orientation of the viewpoint of the operator; and 

a mixed reality image generation step for generating a mixed reality space image 
by superimposing or overlaying the virtual space image on the real space image^ 
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wherein the next position/orientation of the virtual obiect(s) is determined based 
on a location relationship among the position/orientation of the viewpoint of the operaton 
positionfs) of the virtual obiect(s) and the geometric information of the real objectfs) . 

21. (Previously Presented) The method according to claim 20, wherein said 
inputting step captures real space images of said operator's view of the real space, 

and the simulator apparatus comprises a video see-through type display that the 
operator wears wherein the mixed reaUty images are displayed. 

22. (Previously Presented) The method according claim to 20, wherein the 
operator wears an optical see-through type display and the virtual space image is displayed on the 
display. 



23. (Currently Amended) The method according to claim 20, further comprising: 
a status detecting step for detecting a status of the operator, 
wherein said computation step also determines a next action pattern of the virtual 
object in accordance with the status of the operator, ml c stor e d in a ml e m c moiy and in 
coiTespondcnc c with the position/orientation of th e r e al object and/o r the status of the operator, 
and comput e s a position/oii e ntation of th e virtual obj e ct after the d c t c miincd action . 

Claims 24-26 (Canceled) 
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27. (Previously Presented) The method according to claim 20, wherein the real 
object(s) include other operators who operate the simulator apparatus, and wherein the other 
operators share a single mixed reality space with the operator. 



28. (Previously Presented) The method according to claim 20, wherein the real 
object is an object which is fixed in position in the real space, and 

the geometric information acquisition step includes the steps of: 
pre-storing position information and shape information of the real object in a 

predetermined memory; and 

reading out the position information and shape information of the real object from 

the predetermined memory as needed. 

29. (Previously Presented) The method according to claim 20, wherein the real 
object is an object which is movable but does not deform, and the geometric information 
acquisition step includes the steps of: 

pre-storing shape information of the real object in a predetermined memory; 
detecting a position/orientation of the real object by a position/orientation sensor; 

and 

setting a region the real object is expected to occupy in the mixed reality space in 
accordance with the detected position/orientation of the real object. 

Claims 30-36 (Canceled) 
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37. (Previously Presented) The method according to claim 20, wherein the 
viewpoint detection step includes the step of detecting a position/orientation of the head of the 
operator, and 

said method further comprises: 

a detection step of detecting a position/orientation of a hand of the operator; and 
a step of recognizing a relative position of the hand of the operator with respect to 
the head as a command on the basis of an output in the detection step. 

38. (Previously Presented) The method according to claim 20, wherein the 
virtual space generation step includes the steps of: 

aUgning the position/orientation of the real object to the position/orientation of the 
virtual object after movement; and 

generating an image of the virtual object after alignment in correspondence with 
an occlusion relationship. 

39. (Currently Amended) A storage medium which stores a program of an image 
processing method for a simulator apparatus with which an operator plays a simulation in a 
mixed reality space including a virtual space and real space in which a real object(s) is placed, 
comprising: 

a viewpoint detection program step for detecting the position/orientation of a 
viewpoint of the operator; 
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an inputting program step for inputting a real space image corresponding to the 
position/orientation of the viewpoint of the operator; 

a geometric information acquisition program step for acquiring geometric 
information of the real object(s) placed in the real space; 

a computation program step for determining the next action pattern of virtual 
object(s) as if a virtual object had its own will by r c f c iring to ml e s for controlling the action of 
the virtual objcct(s) bas e d on a r e lation among the position/orientation of th e viewpoint of the 
operator, position(s) of th e virtual objcct(s) and the geom e t r ic infoniiation of th e r e al obj e ct(s) 
based on a command input by the operator, a simulation progress status stored in a status 
memorv. rules stored in a rule memory for controlling the action patterns of the virtual obiect(s). 
and the geometric information of the real obiect(s) : 

a virtual space generation program step for generating a virtual space image on the 
basis of ttrc a next position/orientation of the virtual object(s) after according to the action pattern 
and the position/orientation of the viewpoint of the operator; and 

a mixed reality image generation program step for generating a mixed reality 
space image by superimposing or overlaying the virtual space image on the real space image^ 

wherein the next position/orientation of the virtual obiectrs") is determined based 
on a location relationship among the position/orientation of the viewpoint of the operator, 
positionCs) of the virtual object's) and the geometric information of the real objectrs) . 
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